Image processing apparatus and image processing method

ABSTRACT

There is provided an image processing apparatus including an image processing unit which combines a virtual object with a captured image. The image processing unit determines the virtual object based on a state or a type of an object shown in the captured image.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation application of U.S. patentapplication Ser. No. 13/667,225, filed Nov. 2, 2012, which claims thepriority from Japanese Patent Application JP 2011-244377 filed Nov. 8,2011, the entire content of which is hereby incorporated by reference.

BACKGROUND

The present disclosure relates to an image processing apparatus, animage processing method, and a program.

In recent years, a technology has been known that combines variousobjects with an image obtained by capturing (hereinafter, also referredto as “captured image”). Various objects may be combined with a capturedimage, and, for example, in the case where an image of a subject (suchas a person or an animal) is captured, an image of a thing that thesubject has on (such as clothes or a bag) may be combined with thecaptured image as the object. Various technologies are disclosed as thetechnology for combining an object with a captured image.

For example, a technology for combining an image of clothes with animage of a person is disclosed (for example, see JP 2005-136841A). Bylooking at a combined image obtained by combining an image of clotheswith an image of a person, a user can select clothes knowing how theuser would look when the user wears the clothes, even if the user doesnot actually try on the clothes in a shop.

On the other hand, in addition to the technology for combining an objectwith a captured image, there is also a technology for modifying acaptured image itself. As an example of the technology for modifying acaptured image itself, there is given a technology called DiminishedReality. According to the technology, when an object shown in thecaptured image is specified by a user, the specified object is erasedfrom the captured image.

SUMMARY

However, a technology for performing image processing based on a stateor a type of an object shown in a captured image, for example, is notdisclosed. Thus, there was a possibility that a captured image wasprocessed without taking into account a state of an object or a type ofan object. For example, in the case where a user shown in a capturedimage placed clothes on his/her body, if clothes that a user alreadywore came out of the clothes, it was difficult to automatically erasefrom the captured image the part that came out.

Further, it was also difficult to adjust an object to be combined with acaptured image depending on how a user shown in the captured image woreclothes, and to change an object to be combined with a captured imagedepending on what sort of accessory a user shown in the captured imageput on. Therefore, it is desirable that a technology for determining avirtual object to be combined with a captured image based on the stateor the type of an object shown in the captured image is realized.

According to an embodiment of the present disclosure, there is providedan image processing apparatus which includes an image processing unitwhich combines a virtual object with a captured image. The imageprocessing unit determines the virtual object based on a state or a typeof an object shown in the captured image.

According to another embodiment of the present disclosure, there isprovided an image processing method which includes determining, based ona state or a type of an object shown in a captured image, a virtualobject to be combined with the captured image.

According to another embodiment of the present disclosure, there isprovided a program for causing a computer to function as an imageprocessing apparatus, the image processing apparatus including an imageprocessing unit which combines a virtual object with a captured image.The image processing unit determines the virtual object based on a stateor a type of an object shown in the captured image.

According to the embodiments of the present disclosure described above,a virtual object to be combined with a captured image can be determinedbased on the state or the type of an object shown in the captured image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an overview of an image processingsystem according to an embodiment of the present disclosure;

FIG. 2 is a block diagram showing a configuration example of an imageprocessing apparatus;

FIG. 3 is a diagram illustrating skeletal structure information anddepth information;

FIG. 4 is a diagram illustrating an example of a function of an imageprocessing unit;

FIG. 5 is a diagram illustrating an example of a function of the imageprocessing unit;

FIG. 6 is a flowchart showing an example of a flow of operation of theimage processing apparatus;

FIG. 7 is a diagram illustrating a first modified example of a functionof the image processing unit;

FIG. 8 is a diagram illustrating the first modified example of afunction of the image processing unit;

FIG. 9 is a flowchart showing a flow of operation of an image processingapparatus according to the first modified example;

FIG. 10 is a diagram illustrating a second modified example of afunction of the image processing unit;

FIG. 11 is a diagram illustrating the second modified example of afunction of the image processing unit;

FIG. 12 is a diagram illustrating the second modified example of afunction of the image processing unit;

FIG. 13 is a flowchart showing a flow of operation of an imageprocessing apparatus according to the second modified example;

FIG. 14 is a diagram illustrating a third modified example of a functionof the image processing unit;

FIG. 15 is a diagram illustrating the third modified example of afunction of the image processing unit;

FIG. 16 is a diagram illustrating the third modified example of afunction of the image processing unit; and

FIG. 17 is a flowchart showing a flow of operation of an imageprocessing apparatus according to the third modified example.

DETAILED DESCRIPTION OF THE EMBODIMENT(S)

Hereinafter, preferred embodiments of the present disclosure will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the appended drawings, structural elementsthat have substantially the same function and structure are denoted withthe same reference numerals, and repeated explanation of thesestructural elements is omitted.

Further, in this specification and the appended drawings, there are somecases where multiple structural elements that have substantially thesame function and structure are distinguished from one another by beingdenoted with different alphabets after the same reference numeral. Notethat, in the case where it is not necessary to distinguish the multiplestructural elements that have substantially the same function andstructure from one another, the multiple structural elements are denotedwith the same reference numeral only.

Further, the “detailed description of the embodiments” will be describedin the following order.

1. Overview of Image Processing System

2. Functions of Image Processing Apparatus

3. Operation of Image Processing Apparatus

4. Functions in First Modified Example

5. Operation in First Modified Example

6. Functions in Second Modified Example

7. Operation in Second Modified Example

8. Functions in Third Modified Example

9. Operation in Third Modified Example

10. Conclusion

1. OVERVIEW OF IMAGE PROCESSING SYSTEM

In the following, first, an overview of an image processing systemaccording to an embodiment of the present disclosure will be describedwith reference to FIG. 1.

FIG. 1 is a diagram illustrating an overview of an image processingsystem according to the embodiment of the present disclosure. As shownin FIG. 1, an image processing system 10 according to the embodiment ofthe present disclosure includes an image processing apparatus 100, adisplay unit 130, an image capturing unit 140, and a sensor 150. Theplace that the image processing system 10 is to be installed is notparticularly limited. For example, the image processing system 10 may beinstalled at the home of a subject 20.

Also, in the example shown in FIG. 1, a plurality of blocks configuringthe image processing system 10 (for example, the image processingapparatus 100, the display unit 130, the image capturing unit 140, andthe sensor 150) are separately configured, but a combination of some ofthe plurality of blocks configuring the image processing system 10 maybe integrated into one. For example, the plurality of blocks configuringthe image processing system 10 may be embedded in a smartphone, a PDA(Personal Digital Assistant), a mobile phone, a portable musicreproduction device, a portable video processing device, or a portablegame device.

The image capturing unit 140 captures an image of an object existing inthe real space. An object existing in the real space is not particularlylimited, but may be a living thing such as a person or an animal, or athing other than the living thing, such as a garage or a TV table, forexample. In the example shown in FIG. 1, a subject 20 (for example, aperson) is captured by the image capturing unit 140 as the objectexisting in the real space. An image captured by the image capturingunit 140 (hereinafter, also referred to as “captured image”) may bedisplayed by the display unit 130. The captured image to be displayed bythe display unit 130 may be an RGB image. In the example shown in FIG.1, a captured image 131 showing a subject 21 is displayed by the displayunit 130.

The sensor 150 has a function of detecting a parameter in the realspace. For example, in the case where the sensor 150 is configured froman infrared sensor, the sensor 150 can detect infrared radiation in thereal space, and supply, as detection data, an electric signal accordingto the amount of infrared radiation to the image processing apparatus100. The image processing apparatus 100 can recognize the objectexisting in the real space based on the detection data, for example. Thetype of the sensor 150 is not limited to the infrared sensor.Additionally, in the example shown in FIG. 1, the detection data issupplied from the sensor 150 to the image processing apparatus 100, butthe detection data to be supplied to the image processing apparatus 100may also be an image captured by the image capturing unit 140.

A captured image is processed by the image processing apparatus 100. Forexample, the image processing apparatus 100 can process the capturedimage by combining a virtual object with the captured image according tothe recognition result of an object existing in the real space. Thedisplay unit 130 can also display a captured image which has beenprocessed by the image processing apparatus 100. For example, in thecase the position of the subject 21 is recognized by the imageprocessing apparatus 100, a captured image in which a virtual object(such as an image of clothes) is combined at the position of the subject21 may be displayed by the display unit 130. Combining of a virtualobject may be performed by superimposing an image that is registered inadvance and separately from the captured image on the captured image, orby modifying the captured image (for example, by superimposing an imagecaptured from the captured image on the captured image).

By looking at the captured image processed in this manner, the subject20 can select clothes knowing how he/she would look when he/she has wornthe clothes without actually trying the clothes on. However, atechnology for performing image processing corresponding to a state or atype of the subject 20 shown in a captured image, for example, is notdisclosed. Thus, there was a possibility that a virtual object nottaking into account a state or a type of the clothes the subject 20 waswearing was combined with the subject 21. Accordingly, realization of atechnology is desired for determining a virtual object to be combinedwith a captured image based on the state or the type of the object shownin the captured image.

Accordingly, the embodiment of the present disclosure has been attainedwith the above circumstance in mind. According to the embodiment of thepresent disclosure, a virtual object to be combined with a capturedimage can be determined based on the state or the type of the objectshown in the captured image. In the following, functions of the imageprocessing apparatus 100 according to the embodiment of the presentdisclosure will be described with reference to FIGS. 2 to 5.

2. FUNCTIONS OF IMAGE PROCESSING APPARATUS

FIG. 2 is a block diagram showing a configuration example of an imageprocessing apparatus. Referring to FIG. 2, the image processingapparatus 100 includes a control unit 110 and a storage unit 120. Thecontrol unit 110 includes a detection unit 111, an image processing unit112, and a display control unit 113. The display unit 130, the imagecapturing unit 140, and the sensor 150 are connected to the imageprocessing apparatus 100.

(Control Unit)

The control unit 110 corresponds to a processor such as a CPU (CentralProcessing Unit) or a DSP (Digital Signal Processor). The control unit110 causes various functions of the control unit 110 described later tooperate, by executing a program stored in the storage unit 120 or otherstorage media. Additionally, blocks configuring the control unit 110 donot all have to be embedded in the same device, and one or some of themmay be embedded in another device (such as a server).

(Storage Unit)

The storage unit 120 stores a program and data for processing of theimage processing apparatus 100 using a storage medium such as asemiconductor memory or a hard disk. For example, the storage unit 120stores a program for causing a computer to function as the control unit110. Also, for example, the storage unit 120 stores data to be used bythe control unit 110. For example, the storage unit 120 can store afeature quantity dictionary to be used for object recognition andvirtual objects to be displayed.

(Display Unit)

The display unit 130 is a display module that is configured from an LCD(Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), a CRT(Cathode Ray Tube), or the like. In the embodiment of the presentdisclosure, the display unit 130 is assumed to be configured separatelyfrom the image processing apparatus 100, but the display unit 130 may bea part of the image processing apparatus 100.

(Image Capturing Unit)

The image capturing unit 140 generates a captured image by capturing thereal space using an image sensor such as a CCD (Charge Coupled Device)or a CMOS (Complementary Metal Oxide Semiconductor). In the embodimentof the present disclosure, the image capturing unit 140 is assumed to beconfigured separately from the image processing apparatus 100, but theimage capturing unit 140 may be a part of the image processing apparatus100.

(Sensor)

The sensor 150 has a function of detecting a parameter in the realspace. For example, in the case the sensor 150 is configured from aninfrared sensor, the sensor 150 can detect infrared radiation in thereal space, and supply, as detection data, an electric signal accordingto the amount of infrared radiation to the image processing apparatus100. The type of the sensor 150 is not limited to the infrared sensor.Additionally, in the case where an image captured by the image capturingunit 140 is to be supplied to the image processing apparatus 100 as thedetection data, the sensor 150 does not have to exist.

(Detection Unit)

The detection unit 111 detects the state or the type of an object shownin a captured image based on detection data. Examples of the state orthe type of the object to be detected by the detection unit 111 will bedescribed later, and the state or the type of the object to be detectedby the detection unit 111 is not particularly limited. Further, a methodof detecting the state or the type of the object shown in the capturedimage is not particularly limited. For example, the detection unit 111matches the feature quantity determined from the captured image againstthe feature quantity of an object, and can thereby recognize the stateor the type of the object included in the captured image.

More specifically, the detection unit 111 determines a feature quantityin a captured image according to a feature quantity determination methodsuch as SIFT method or Random Ferns method, and matches the determinedfeature quantity against the feature quantity of an object. Then, thedetection unit 111 recognizes the type of the object associated with afeature quantity that best matches the feature quantity in the capturedimage (for example, information which is for identifying the object),and the state of the object in the captured image (for example, theposition and the attitude of the object).

Here, a feature quantity dictionary in which feature quantity data ofobjects and information for identifying types of objects are associatedis used by the detection unit 111. This feature quantity dictionary maybe stored in the storage unit 120, or may be received from a server. Thefeature quantity data of an object may be a collection of featurequantities determined from images for learning of an object according toSIFT method or Random Ferns method, for example.

Alternatively, the detection unit 111 can also detect the state or thetype of the object by matching a feature quantity determined from thedetection data supplied from the sensor 150, instead of the capturedimage, against the feature quantity of the object. In this case, forexample, the detection unit 111 can detect the state or the type of theobject based on the detection data supplied from the sensor 150 using amethod similar to the method of detecting the state or the type of theobject from the captured image.

The state or the type of an object shown in a captured image can bedetected based on detection data by a method as described above. Thedetection unit 111 may also be integrated not in the image processingapparatus 100, but in the sensor 150.

As in the case of recognizing an object, the detection unit 111 canrecognize a part of an object included in a captured image by matching afeature quantity determined from the captured image against a featurequantity of each part of the object. Alternatively, the detection unit111 can recognize a part of an object by matching a feature quantitydetermined from the detection data supplied from the sensor 150, insteadof the captured image, against the feature quantity of each part of theobject.

A known art (for example, Kinect (registered trademark) developed byMicrosoft Corporation (registered trademark)) can be used to detect thestate of an object. By using such a known art, the detection unit 111can acquire skeletal structure information as an example of coordinatesshowing the position of each of one or more parts forming the subject20. The detection unit 111 can detect the state of the subject 20 basedon the skeletal structure information. Alternatively, the detection unit111 can detect the state of the subject 20 based on the skeletalstructure information and depth information. First, the skeletalstructure information and the depth information will be described withreference to FIG. 3.

FIG. 3 is a diagram illustrating skeletal structure information anddepth information. The detection unit 111 can acquire skeletal structureinformation as shown in FIG. 3 by using the known art described above.In the example shown in FIG. 3, the skeletal structure information isshown as coordinates B1 to B3, B6, B7, B9, B12, B13, B15, B17, B18, B20to B22, and B24 showing 15 parts forming the subject 20, but the numberof parts included in the skeletal structure information is notparticularly limited.

Additionally, the coordinates B1 represent the coordinates of “Head”,the coordinates B2 represent the coordinates of “Neck”, the coordinatesB3 represent the coordinates of “Torso”, the coordinates B6 representthe coordinates of “Left Shoulder”, and the coordinates B7 represent thecoordinates of “Left Elbow”. Also, the coordinates B9 represent thecoordinates of “Left Hand”, the coordinates B12 represent thecoordinates of “Right Shoulder”, the coordinates B13 represent thecoordinates of “Right Elbow”, and the coordinates B15 represent thecoordinates of “Right Hand”.

The coordinates B17 represent the coordinates of “Left Hip”, thecoordinates B18 represent the coordinates of “Left Knee”, thecoordinates B20 represent the coordinates of “Left Foot”, and thecoordinates B21 represent the coordinates of “Right Hip”. Thecoordinates B22 represent the coordinates of “Right Knee”, and thecoordinates B24 represent the coordinates of “Right Foot”.

Further, the detection unit 111 can acquire depth information as shownin FIG. 3 by using the known art described above. The depth informationis information indicating the distance from the sensor 150, and, in FIG.3, an object existing region R where the depth is below a thresholdvalue (a region where the distance from the sensor 150 is less than thethreshold value) and other region R′ are shown as examples of the depthinformation for the sake of simplicity.

(Image Processing Unit)

The image processing unit 112 processes a captured image based on thestate or the type of an object shown in the captured image. The state orthe type of an object shown in the captured image may be detected by thedetection unit 111 as described above. No particular limitation isimposed as to how the image processing unit 112 processes the capturedimage. For example, as described above, the image processing unit 112can process the captured image by combining a virtual object with thecaptured image based on the state or the type of an object shown in thecaptured image. As described above, combining of a virtual object may beperformed by superimposing an image that is registered in advance andseparately from the captured image on the captured image, or bymodifying the captured image (for example, by superimposing an imagecaptured from the captured image on the captured image).

Subsequently, an example of a function of the image processing unit 112will be described with reference to FIG. 4 and FIG. 5. FIG. 4 is adiagram illustrating an example of a function of the image processingunit 112.

For example, in the case where the adjacency of an object shown in acaptured image and a predetermined object shown in the captured image isdetected, the image processing unit 112 may combine a virtual objectwith the captured image by processing of replacing the object shown inthe captured image with another object. FIG. 4 shows an object 31A as anexample of the object shown in a captured image 131A, and an object 32Aas an example of the predetermined object shown in the captured image131A. Further, FIG. 4 shows an object 33A as an example of the otherobject. Further, in the example shown in FIG. 4, the object 31A is askirt, the object 32A is a pair of trousers, and the object 33A is aleg. However, the object 31A, the object 32A, and the object 33A are notparticularly limited.

In the captured image 131A, the subject 21 wears the trousers, andplaces the skirt on the trousers. By looking at the displayed capturedimage 131A, the subject 20 can know how he/she would look when he/shehas worn the skirt. However, in the captured image 131A, the trouserscome out from the bottom of the skirt, but on the other hand, when thesubject 20 actually wears the skirt, his/her legs are supposed to comeout from the bottom of the skirt. Thus, the subject 20 grasps, from thecaptured image 131A, an appearance different from how he/she would lookwhen he/she has actually worn the skirt.

Accordingly, in the case where the adjacency of the skirt and thetrousers is detected, the image processing unit 112 may replace thetrousers with legs (for example, skin-colored part). FIG. 4 shows acaptured image 131A′ in which an excess part of the trousers that comesout of the skirt is erased and the part is replaced with the legs by theimage processing unit 112. The adjacency of the skirt and the trousersis detected by the detection unit 111, and the detection method is notparticularly limited.

For example, the adjacency of the object shown in a captured image andthe predetermined object may be detected based on at least one of thecolor or the shape of the object that is adjacent to the object shown inthe captured image. For example, the detection unit 111 determineswhether the shape of an excess part that comes out of the skirt is a legshape, and in the case of determining that the excess part is not theleg shape, since it is the presumed that trousers are coming out of theskirt, the adjacency of the skirt and the trousers may be detected.

Further, for example, the detection unit 111 determines whether thecolor of the excess part that comes out of the skirt is a leg color (forexample, skin color), and in the case of determining that the color ofthe excess part is not the leg color, since it is presumed that thetrousers are coming out of the skirt, the adjacency of the skirt and thetrousers may be detected. In addition, even though the detection unit111 determines that the color of the excess part that comes out of theskirt is the leg color, in the case where the excess part with the legcolor does not have a size exceeding a threshold, since it is notpresumed that the leg-colored part is a leg, the adjacency of the skirtand the trousers may be detected.

In this way, the detection unit 111 can detect the adjacency of theskirt and the trousers based on the shape or the color of the excesspart that comes out of the skirt. The leg shape or the leg color may beregistered in advance, or may be accumulated by learning. Note that, inthe case where the adjacency of an object shown in a captured image anda predetermined object superimposed on the captured image is detected,the image processing unit 112 may combine a virtual object with thecaptured image by processing of replacing the object shown in thecaptured image with another object. In the example shown in FIG. 4, thesubject 20 places the skirt existing in a real space on his/her body,and also in the case where a skirt, which is a virtual object, is placedon the body of the subject 21, the processing of erasing the excess partof the trousers that comes out of the skirt may be executed.

FIG. 5 is a diagram illustrating an example of a function of the imageprocessing unit 112. FIG. 5 shows an object 31B as an example of theobject shown in a captured image 131B, and an object 32B as an exampleof the predetermined object shown in the captured image 131B. Further,FIG. 5 shows an object 33B as an example of the other object. Further,in the example shown in FIG. 5, the object 31B is a short-sleeved shirt,the object 32B is a long-sleeved shirt, and the object 33B is an arm.However, the object 31B, the object 32B, and the object 33B are notparticularly limited.

In the captured image 131B, the subject 21 wears the long-sleeved shirt,and places the short-sleeved shirt on the long-sleeved shirt. By lookingat the displayed captured image 131B, the subject 20 can know how he/shewould look when he/she has worn the short-sleeved shirt. However, in thecaptured image 131B, the long-sleeved shirt comes out from under theshort-sleeved shirt, but on the other hand, when the subject 20 actuallywears the short-sleeved shirt, his/her arms are supposed to come outfrom under the short-sleeved shirt. Thus, the subject 20 grasps, fromthe captured image 131B, an appearance different from how he/she wouldlook when he/she has actually worn the short-sleeved shirt.

Accordingly, in the case where the adjacency of the long-sleeved shirtand the short-sleeved shirt is detected, the image processing unit 112may replace the long-sleeved shirt with arms (for example, skin-coloredpart). FIG. 5 shows a captured image 131B′ in which an excess part ofthe long-sleeved shirt that comes out of the short-sleeved shirt iserased and the excess part is replaced with the arms by the imageprocessing unit 112. The adjacency of the long-sleeved shirt and theshort-sleeved shirt is detected by the detection unit 111, and thedetection method is not particularly limited. For example, the detectionunit 111 determines whether the shape of an excess part that comes outof the short-sleeved shirt is an arm shape, and in the case ofdetermining that the excess part is not the arm shape, since it ispresumed that the long-sleeved shirt is coming out of the short-sleevedshirt, the adjacency of the long-sleeved shirt and the short-sleevedshirt may be detected.

Further, for example, the detection unit 111 determines whether thecolor of the excess part that comes out of the short-sleeved shirt is anarm color (for example, skin color), and in the case of determining thatthe color of the excess part is not the arm color, since it is presumedthat the long-sleeved shirt are coming out of the short-sleeved shirt,the adjacency of the short-sleeved shirt and the long-sleeved shirt maybe detected. In addition, even though the detection unit 111 determinesthat the color of the excess part that comes out of the short-sleevedshirt is the arm color, in the case where the excess part with the armcolor does not have a size exceeding a threshold, since it is notpresumed that the arm-colored part is an arm, the adjacency of theshort-sleeved shirt and the long-sleeved shirt may be detected.

In this way, the detection unit 111 can detect the adjacency of theshort-sleeved shirt and the long-sleeved shirt based on the shape or thecolor of the excess part that comes out of the short-sleeved shirt. Thearm shape or the arm color may be registered in advance, or may beaccumulated by learning. Note that, in the example shown in FIG. 5, thesubject 20 places the short-sleeved shirt existing in a real space onhis/her body, and also in the case where a short-sleeved shirt, which isa virtual object, is placed on the body of the subject 21, theprocessing of erasing the excess part of the long-sleeved shirt thatcomes out of the short-sleeved shirt may be executed.

Note that, even if an excess part that comes out of clothes is present,there is also a case where it is preferred that the part be not erased.For example, it is represented by the following case: when the subject20 wears a jacket on top of a shirt which he/she has already worn, thesubject 20 wants to know how he/she would look in a state without theshirt being erased. In order to satisfy such a requirement, for example,in the case where the object shown in the captured image is adjacent tothe predetermined object shown in the captured image at a predeterminedposition, the predetermined object shown in the captured image may notbe replaced with the other object.

For example, if the shirt comes out of the jacket and the excess partthat comes out of the jacket is a neck of the subject 21, the imageprocessing unit 112 may not replace the predetermined object shown inthe captured image with the neck (for example, skin-colored part).Accordingly, in the case where an adjacent position is a predeterminedposition (for example, arm position or leg position), the imageprocessing unit 112 may replace the object that is adjacent to theobject shown in the captured image with the other object, and in thecase where the adjacent position is another position (for example, neckposition), the image processing unit 112 may not replace the object thatis adjacent to the object shown in the captured image with the otherobject. The adjacent position may be determined based on the capturedimage, or may be determined based on the skeletal structure information.

(Display Control Unit)

The display unit 130 is controlled by the display control unit 113 suchthat a captured image with which the virtual object has been combined bythe image processing unit 112 is displayed on the display unit 130.

Heretofore, the examples of the functions of the image processingapparatus 100 according to the embodiment of the present disclosure havebeen described with reference to FIG. 4 and FIG. 5. In the following, anexample of a flow of operation of the image processing apparatus 100according to the embodiment of the present disclosure will be describedwith reference to FIG. 6.

3. OPERATION OF IMAGE PROCESSING APPARATUS

FIG. 6 is a flowchart showing an example of a flow of operation of theimage processing apparatus 100 according to the embodiment of thepresent disclosure. Note that the operation of the image processingapparatus 100 to be described with reference to FIG. 6 is particularlyoperation of the image processing apparatus 100 in the case of detectinga state of clothes that the subject 20 wears. The clothes that thesubject 20 wears are an example of the object shown in the capturedimage.

As shown in FIG. 6, first, in the case where the subject 21 does notplace clothes on his/her body (“No” in Step S11), the detection unit 111returns to Step S11. On the other hand, in the case where the subject 21places clothes on his/her body (“Yes” in Step S11), the detection unit111 detects the state of the clothes. In the case where it is detectedthat other clothes come out of the clothes placed on the body at thearms of the subject 20 (“Yes” in Step S12), the image processing unit112 erases an excess part of the other clothes that comes out of theclothes placed on the body at the arms (Step S13).

On the other hand, in the case where it is not detected that the otherclothes come out of the clothes placed on the body at the arms of thesubject 20 (“No” in Step S12), the image processing unit 112 proceeds toStep S14. In the case where it is detected that other clothes come outof the clothes placed on the body at the legs of the subject 20 (“Yes”in Step S14), the image processing unit 112 erases an excess part of theother clothes that comes out of the clothes placed on the body at thelegs (Step S15).

On the other hand, in the case where it is not detected that the otherclothes come out of the clothes placed on the body at the legs of thesubject 20 (“No” in Step S14), the image processing unit 112 may noterase the excess part of the other clothes that comes out of the clothesplaced on the body at the legs. The display unit 130 may be controlledby the display control unit 113 such that a captured image processed bythe image processing unit 112 is displayed on the display unit 130.

Heretofore, the example of the flow of operation of the image processingapparatus 100 has been described with reference to FIG. 6.

4. FUNCTIONS IN FIRST MODIFIED EXAMPLE

Subsequently, a first modified example of functions of the imageprocessing unit 112 will be described with reference to FIG. 7 and FIG.8. FIG. 7 is a diagram illustrating a first modified example of afunction of the image processing unit 112.

For example, in the case where an overlap between an object shown in acaptured image and a predetermined object shown in the captured image isdetected, the image processing unit 112 may combine, with the capturedimage, a virtual object modified in accordance with the overlap. FIG. 7shows an object 31C as an example of the object shown in a capturedimage 131C, and an object 32C as an example of the predetermined objectshown in the captured image 131C. Further, in the example shown in FIG.7, the object 31C is a shirt, the object 32C is a pair of trousers, andthe object 33C is a pair of trousers. However, the object 31C, theobject 32C, and the object 33C are not particularly limited.

In the captured image 131C, the subject 21 wears the trousers and theshirt. By looking at the captured image in which the trousers arecombined with the captured image 131C, the subject 20 can know howhe/she would look when he/she has worn the trousers. However, if thewhole pair of trousers is uniformly combined with the captured image131C regardless of whether the shirt hangs out of the trousers, it isdifficult to obtain a captured image that naturally reflects a manner inwhich the subject 20 wears the trousers. Thus, the subject 20 grasps anappearance of the case of wearing the trousers different from the mannerof the way the subject 20 would wear the trousers.

Accordingly, in the case where an overlap between the shirt and thetrousers is detected, the image processing unit 112 may combine, withthe captured image, the trousers modified in accordance with theoverlap. FIG. 7 shows a captured image 131C′ in which the whole pair oftrousers is combined with the captured image 131C, since the overlapbetween the shirt and the trousers is not detected by the imageprocessing unit 112. The overlap between the shirt and the trousers isdetected by the detection unit 111, and the detection method is notparticularly limited.

For example, the overlap between the object shown in a captured imageand the predetermined object may be detected based on a position of aboundary line between the object shown in the captured image and thepredetermined object. For example, the detection unit 111 determineswhether the position of the boundary line between the shirt and thetrousers is a position of the waist of the subject 21, and in the caseof determining that the position of the boundary line is the position ofthe waist of the subject 21, since it is presumed that the subject 20tucks the shirt in the trousers, the overlap between the shirt and thetrousers may be detected. Among the objects shown in the captured image,the detection unit 111 may regard a white area as a shirt area, and mayregard the bottom edge of the shirt area as the boundary line betweenthe shirt and the trousers. The position of the waist may be determinedbased on the captured image, or may be determined based on the skeletalstructure information. Further, the position of the waist may alsoinclude the position in the vicinity of the waist.

Further, for example, the detection unit 111 may detect the overlapbetween the shirt and the trousers in the case where the position of theboundary line is the position of the waist of the subject 21 and theboundary line is similar to a straight line. For example, the detectionunit 111 may detect the overlap between the shirt and the trousers inthe case where the bottom edge of the shirt area is approximated bymultiple points and the line obtained by connecting the multiple pointsis similar to the straight line. The range of similarity can be set inadvance. In this way, the detection unit 111 can detect the overlapbetween the shirt and the trousers based on the position of the boundaryline between the shirt and the trousers.

FIG. 8 is a diagram illustrating a modified example of a function of theimage processing unit 112. FIG. 8 shows an object 31D as an example ofthe object shown in a captured image 131D, and an object 32D as anexample of the predetermined object shown in the captured image 131D.Further, in the example shown in FIG. 8, the object 31D is a shirt, theobject 32D is a pair of trousers, and the object 33D is a pair oftrousers. However, the object 31D, the object 32D, and the object 33Dare not particularly limited.

As described above, in the case where the overlap between the shirt andthe trousers is detected, the image processing unit 112 may combine,with the captured image, the trousers modified in accordance with theoverlap. FIG. 8 shows a captured image 131D′ in which a part of thetrousers is combined with the captured image 131C, since the overlapbetween the shirt and the trousers is detected by the image processingunit 112. The overlap between the shirt and the trousers is detected bythe detection unit 111, and the detection method is not particularlylimited. The example of the detection method is as described above.

Heretofore, the first modified example of the functions of the imageprocessing apparatus 100 according to the embodiment of the presentdisclosure has been described with reference to FIG. 7 and FIG. 8. Inthe following, an example of a flow of operation of the image processingapparatus 100 according to the first modified example will be describedwith reference to FIG. 9.

5. OPERATION IN FIRST MODIFIED EXAMPLE

FIG. 9 is a flowchart showing a flow of operation of the imageprocessing apparatus 100 according to the first modified example. Notethat the operation of the image processing apparatus 100 to be describedwith reference to FIG. 9 is particularly operation of the imageprocessing apparatus 100 in the case of detecting a state of a shirtthat the subject 20 wears. The shirt that the subject 20 wears is anexample of the object shown in the captured image.

As shown in FIG. 9, first, in the case of not detecting the shirt (“No”in Step S21), the detection unit 111 returns to Step S21. On the otherhand, in the case of detecting the shirt (“Yes” in Step S21), thedetection unit 111 detects the state of the shirt. In the case where itis determined that the subject 21 tucks the shirt in the trousers (“Yes”in Step S22), the image processing unit 112 superimposes the whole pairof trousers on the captured image (Step S23).

On the other hand, in the case where it is detected that the subject 21wears the shirt in a state the shirt hanging out of the trousers (“No”in Step S22), the image processing unit 112 superimposes a part of thetrousers that is not overlapped with the shirt on the captured image(Step S24). The display unit 130 may be controlled by the displaycontrol unit 113 such that a captured image processed by the imageprocessing unit 112 is displayed on the display unit 130.

Heretofore, the flow of operation of the image processing apparatus 100according to the first modified example has been described withreference to FIG. 9.

6. FUNCTIONS IN SECOND MODIFIED EXAMPLE

Subsequently, a second modified example of functions of the imageprocessing unit 112 will be described with reference to FIGS. 10 to 12.FIG. 10 is a diagram illustrating a second modified example of afunction of the image processing unit 112.

For example, in the case where a type of an object shown in a capturedimage is detected, the image processing unit 112 may combine a virtualobject with the captured image, by processing of combining, with thecaptured image, the virtual object selected in accordance with the typeof the object shown in the captured image. Further, in the case wherethe type of the object shown in the captured image is detected, theimage processing unit 112 may combine the virtual object with thecaptured image, by modifying another object shown in the captured imageinto a shape corresponding to the type of the object shown in thecaptured image.

FIG. 10 shows an object 31E as an example of the object shown in acaptured image 131E, and an object 32E as an example of the other objectshown in the captured image 131E. Further, FIG. 10 shows an object 33Eas an example of the other object. In the example shown in FIG. 10, theobject 31E is a necklace having a short length in a vertical direction(for example, necklace having a length x₁ in the vertical direction,which is shorter than a threshold), the object 32E is clothes that thesubject 21 wears, and the object 33E is round neck clothes. However, theobject 31E, the object 32E, and the object 33C are not particularlylimited.

In the captured image 131E, the subject 21 wears the necklace having ashort length in the vertical direction. By looking at the displayedcaptured image 131E, the subject 20 can know how he/she would look whenhe/she has worn the necklace having a short length in the verticaldirection. However, in the case where the subject 20 tries on a necklacein a shop, for example, the subject 20 may not be wearing clothessuitable for the necklace. Thus, the subject 20 grasps, from thecaptured image 131E, an appearance different from how he/she would lookwhen he/she has changed into clothes suitable for the necklace.

Accordingly, the image processing unit 112 may modify the shape of theclothes in accordance with the detected type of an accessory. FIG. 10shows a captured image 131E′ in which the shape of the clothes ischanged to the round neck by the image processing unit 112, since thenecklace having a short length in the vertical direction has beendetected. The type of the accessory is detected by the detection unit111, and the detection method is not particularly limited. For example,the type of the accessory may be detected by a method of recognizing thetype of an object as described above.

The method of changing the shape of the clothes is also not particularlylimited. For example, in the case where a necklace having a short lengthin the vertical direction is detected, the image processing unit 112 maychange the shape of the clothes by processing of combining, with thecaptured image, round neck clothes that is suited to the necklace havinga short length in the vertical direction. Further, for example, in thecase where the necklace having a short length in the vertical directionis detected, the image processing unit 112 may change the shape of theclothes by modifying the clothes shown in the captured image 131E intothe round neck shape, which is a shape that is suited to the necklacehaving a short length in the vertical direction.

Note that, although the subject 20 wears the accessory existing in thereal space in the example shown in FIG. 10, processing of changing theshape of the clothes may also be executed even in the case where anaccessory serving as a virtual object is superimposed on the subject 21.

FIG. 11 is a diagram illustrating the second modified example of afunction of the image processing unit 112. FIG. 11 shows an object 31Fas an example of the object shown in a captured image 131F, and anobject 32F as an example of the other object shown in the captured image131F. Further, FIG. 11 shows an object 33F as an example of the otherobject. In the example shown in FIG. 11, the object 31F is a necklacehaving a long length in the vertical direction (for example, necklacehaving a length x₂ in the vertical direction, which is longer than athreshold), the object 32F is clothes that the subject 21 wears, and theobject 33F is V-neck clothes. However, the object 31F, the object 32F,and the object 33F are not particularly limited.

FIG. 11 shows a captured image 131F′ in which the shape of the clotheshas been changed to the V-neck by the image processing unit 112, sincethe necklace having a long length in the vertical direction has beendetected. For example, in the case where a necklace having a long lengthin the vertical direction is detected, the image processing unit 112 maychange the shape of the clothes by processing of combining, with thecaptured image, V-neck clothes that are suited to the necklace having along length in the vertical direction. Further, for example, in the casewhere the necklace having a long length in the vertical direction isdetected, the image processing unit 112 may change the shape of theclothes by modifying the clothes shown in the captured image 131F intothe V-neck shape, which is a shape that is suited to the necklace havinga long length in the vertical direction.

FIG. 12 is a diagram illustrating the second modified example of afunction of the image processing unit 112. FIG. 12 shows an object 31Gas an example of the object shown in a captured image 131G, and anobject 32G as an example of the other object shown in the captured image131F. Further, FIG. 12 shows an object 33G as an example of the otherobject. In the example shown in FIG. 12, the object 31G is an earring,the object 32G is clothes that the subject 21 wears, and the object 33Gis boat neck clothes. However, the object 31G, the object 32G, and theobject 33G are not particularly limited.

FIG. 12 shows a captured image 131G′ in which the shape of the clotheshas been changed to the boat neck by the image processing unit 112,since the earrings are detected. For example, in the case where earringsare detected, the image processing unit 112 may change the shape of theclothes by processing of combining, with the captured image, boat neckclothes that are suited to the earrings. Further, for example, in thecase where the earrings are detected, the image processing unit 112 maychange the shape of the clothes by modifying the clothes shown in thecaptured image 131G into the boat neck shape, which is a shape that issuited to the earrings.

Heretofore, the second modified example of the functions of the imageprocessing apparatus 100 according to the embodiment of the presentdisclosure has been described with reference to FIGS. 10 to 12. In thefollowing, an example of a flow of operation of the image processingapparatus 100 according to the second modified example will be describedwith reference to FIG. 13.

7. OPERATION IN SECOND MODIFIED EXAMPLE

FIG. 13 is a flowchart showing a flow of operation of the imageprocessing apparatus 100 according to the second modified example. Notethat the operation of the image processing apparatus 100 to be describedwith reference to FIG. 13 is particularly operation of the imageprocessing apparatus 100 in the case of detecting a type of an accessorythat the subject 20 wears. The accessory that the subject 20 wears is anexample of the object shown in the captured image.

As shown in FIG. 13, first, the detection unit 111 detects the type ofthe accessory shown in the captured image. In the case of detecting thenecklace from the captured image (“Yes” in Step S31), the detection unit111 detects the length of the necklace in the vertical direction. In thecase where the detection unit 111 detects that the length of thenecklace in the vertical direction is short (“Yes” in Step S32), theimage processing unit 112 changes the shape of the clothes shown in thecaptured image to the round neck (Step S33).

On the other hand, in the case where the detection unit 111 detects thatthe length of the necklace in the vertical direction is long (“No” inStep S32), the image processing unit 112 changes the shape of theclothes shown in the captured image to the V-neck (Step S34). In thecase where the necklace is not detected on the captured image by thedetection unit 111 (“No” in Step S31), and earrings are detected fromthe captured image (“Yes” in Step S35), the image processing unit 112changes the shape of the clothes shown in the captured image to the boatneck (Step S36). The display unit 130 may be controlled by the displaycontrol unit 113 such that a captured image processed by the imageprocessing unit 112 is displayed on the display unit 130.

Heretofore, the flow of operation of the image processing apparatus 100according to the second modified example has been described withreference to FIG. 13.

8. FUNCTIONS IN THIRD MODIFIED EXAMPLE

Subsequently, a third modified example of functions of the imageprocessing unit 112 will be described with reference to FIGS. 14 to 16.FIG. 14 is a diagram illustrating a third modified example of a functionof the image processing unit 112.

FIG. 14 shows an object 31H as an example of the object shown in acaptured image 131H. Further, FIG. 14 shows an object 33H as an exampleof the other object. In the example shown in FIG. 14, the object 31H isboat neck clothes, and the object 33H is an earring. However, the object31H and the object 33H are not particularly limited.

In the captured image 131H, although the subject 21 wears the clothes,the subject 21 does not wear any accessories. For example, if anaccessory suitable for the clothes that the subject 21 wears is combinedwith the captured image 131H, the subject 20 can know the accessorysuitable for the clothes that he/she wears by looking at the capturedimage with which the accessory is combined. Further, it is highly likelythat the subject 20 may like the accessory.

Accordingly, the image processing unit 112 may change the type of theaccessory to be combined with the captured image depending on thedetected type of clothes. FIG. 14 shows a captured image 131H′ in whichearrings are combined by the image processing unit 112, since the boatneck clothes are detected. It is more preferred that the earrings to becombined here be hanging down earrings. This is because it is expectedthat the balance as a whole becomes better by making it look long in thevertical direction using the earrings. The type of the clothes isdetected by the detection unit 111, and the detection method is notparticularly limited. For example, the type of the clothes may bedetected by a method of recognizing the type of an object as describedabove.

Note that, although the subject 20 wears the clothes existing in thereal space in the example shown in FIG. 14, processing of changing thetype of the accessory may also be executed even in the case where theclothes serving as a virtual object is superimposed on the subject 21.

FIG. 15 is a diagram illustrating the third modified example of afunction of the image processing unit 112. FIG. 15 shows an object 31Ias an example of the object shown in a captured image 131I. Further,FIG. 15 shows an object 33I as an example of the other object. In theexample shown in FIG. 15, the object 31I is V-neck clothes, and theobject 33I is a necklace having a long length in the vertical direction.However, the object 31I and the object 33I are not particularly limited.

FIG. 15 shows a captured image 131I′ in which the necklace having a longlength in the vertical direction is combined by the image processingunit 112, since the V-neck clothes is detected.

FIG. 16 is a diagram illustrating the third modified example of afunction of the image processing unit 112. FIG. 16 shows an object 31Jas an example of the object shown in a captured image 131J. Further,FIG. 16 shows an object 33J as an example of the other object. In theexample shown in FIG. 16, the object 31J is round neck clothes, and theobject 33J is a necklace having a short length in the verticaldirection. However, the object 31J and the object 33J are notparticularly limited.

FIG. 16 shows a captured image 131J′ in which the necklace having ashort length in the vertical direction is combined by the imageprocessing unit 112, since the round neck clothes is detected.

9. OPERATION IN THIRD MODIFIED EXAMPLE

FIG. 17 is a flowchart showing a flow of operation of the imageprocessing apparatus 100 according to the third modified example. Notethat the operation of the image processing apparatus 100 to be describedwith reference to FIG. 17 is particularly operation of the imageprocessing apparatus 100 in the case of detecting a type of clothes thatthe subject 20 wears. The clothes that the subject 20 wears are anexample of the object shown in the captured image.

As shown in FIG. 17, first, the detection unit 111 detects the type ofthe clothes shown in the captured image. In the case where the detectionunit 111 detects that the shape of the clothes is the boat neck (“Yes”in Step S41), the image processing unit 112 superimposes earrings on thecaptured image (Step S33). On the other hand, in the case where thedetection unit 111 detects that the shape of the clothes is not the boatneck (“No” in Step S41), the image processing unit 112 proceeds to StepS43.

In the case where the detection unit 111 detects that the shape of theclothes is the V-neck (“Yes” in Step S43), the image processing unit 112superimposes a necklace having a long length in the vertical directionon the captured image (Step S44). On the other hand, in the case wherethe detection unit 111 detects that the shape of the clothes is not theV-neck (“No” in Step S43), the image processing unit 112 proceeds toStep S45.

In the case where the detection unit 111 detects that the shape of theclothes is the round neck (“Yes” in Step S45), the image processing unit112 superimposes a necklace having a short length in the verticaldirection on the captured image (Step S46). The display unit 130 may becontrolled by the display control unit 113 such that a captured imageprocessed by the image processing unit 112 is displayed on the displayunit 130. Heretofore, the flow of operation of the image processingapparatus 100 according to the first modified example has been describedwith reference to FIG. 17.

10. CONCLUSION

As described above, according to the embodiments of the presentdisclosure, an image processing apparatus is provided which includes animage processing unit for combining a virtual object with a capturedimage, where the image processing unit determines the virtual objectbased on the state or the type of an object shown in the captured image.According to such an image processing apparatus, a virtual object can bedetermined taking into account the state or the type of an object shownin a captured image. According to such an image processing apparatus, anexcess part of clothes shown in the captured image that comes out ofother clothes can be erased, and new clothes can be combined in a mannerto reflect the way a user wears clothes, for example. Further, anaccessory that matches the clothes that the user wears can be combinedwith the captured image, and the clothes can be changed to the shapethat matches the accessory worn by the user.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

For example, the image processing apparatus 100 including the imageprocessing unit 112 may be provided in a server or in a terminal capableof communication with a server. Also, an example has been mainlydescribed above where the function of detecting the state or the type ofan object is provided in the image processing apparatus 100, but such afunction may be provided in a device other than the image processingapparatus 100. For example, such a function may be provided in thesensor 150. For example, in the case where the image processingapparatus 100 transmits a captured image to another device, the otherdevice may detect the state or the type of an object based on thecaptured image, instead of the image processing apparatus 100.

Further, for example, an example has been mainly described above wherethe display control unit 113 is provided in the image processingapparatus 100, but the display control unit 113 may be provided in adevice other than the image processing apparatus 100. For example, theimage processing unit 112 may be provided in a server, and the displaycontrol unit 113 may be provided in a terminal. For example, in the casewhere a captured image processed by a server is transmitted to aterminal, the terminal may control the display unit 130 such that thecaptured image is displayed by the display unit 130. In this manner, thetechnology of the present disclosure can be applied also to cloudcomputing.

Further, for example, in the third modified example, an example has beenmainly described where the image processing unit 112 changes the shapeof the clothes in accordance with the type of the accessory, and changesthe type of the accessory in accordance with the shape of the clothes.That is, the virtual object is determined taking into account thecompatibility of the type of the accessory with the shape of theclothes. However, in addition to the combination of the accessory andthe shape of the clothes, there are combinations which influence thecompatibility. For example, the combination of the length of a skirt andthe length of boots influences the compatibility. Accordingly, the imageprocessing unit 112 can also determine the virtual object by taking intoaccount the compatibility of the length of the skirt with the length ofthe boots. In the same manner, the image processing unit 112 can alsodetermine the virtual object by taking into account the compatibility ofthe height of shoes (for example, height of heels and length of boots)with the length of trousers.

Further, in the case where the detection unit 111 detects a belt, theimage processing unit 112 may perform image processing in a manner thata waist part of a one-piece dress that is worn with the belt by asubject is tightened. The one-piece dress worn by the subject may besuperimposed on the captured image, or may be shown in the capturedimage. If such image processing is performed, the subject can know howhe/she would look when he/she has worn the one-piece dress with thebelt, without actually performing, by the subject, the operation oftightening the one-piece dress with the belt. Further, the imageprocessing unit 112 may also determine the virtual object taking intoaccount the compatibility of clothes. For example, in the case where thewearing of a muffler is detected, the image processing unit 112 maydetermine the virtual object in a manner to change the sleeves of theclothes that is worn with the muffler from the short sleeves to the longsleeves. If such image processing is performed, even when the season inwhich the subject is trying on the clothes is different from the seasonin which the subject actually wears the clothes, for example, thesubject can know more accurately how he/she would look when he/she hasworn the clothes.

Further, respective steps included in the operation of the imageprocessing apparatus 100 of the present specification are notnecessarily processed in chronological order in accordance with theflowcharts. For example, the respective steps included in the operationof the image processing apparatus 100 may be processed in differentorder from the flowcharts, or may be processed in a parallel manner.

Further, it is also possible to create a computer program for causinghardware such as a CPU, a ROM, and a RAM, which are built in the imageprocessing apparatus 100, to exhibit equivalent functions as those ofrespective structures of the image processing apparatus 100 describedabove. Further, there is also provided a storage medium having thecomputer program stored therein. Additionally, the present technologymay also be configured as below.

(1) An image processing apparatus including

an image processing unit which combines a virtual object with a capturedimage,

wherein the image processing unit determines the virtual object based ona state or a type of an object shown in the captured image.

(2) The image processing apparatus according to (1),

wherein, in a case where an adjacency of an object shown in the capturedimage and a predetermined object shown in the captured image isdetected, the image processing unit combines a virtual object with thecaptured image by processing of replacing the object shown in thecaptured image with another object.

(3) The image processing apparatus according to (1),

wherein, in a case where an adjacency of an object shown in the capturedimage and a predetermined object superimposed on the captured image isdetected, the image processing unit combines a virtual object with thecaptured image by processing of replacing the object shown in thecaptured image with another object.

(4) The image processing apparatus according to (2) or (3),

wherein the adjacency of the object shown in the captured image and thepredetermined object is detected based on at least one of a color or ashape of an object that is adjacent to the object shown in the capturedimage.

(5) The image processing apparatus according to (1),

wherein, in a case where an overlap between an object shown in thecaptured image and a predetermined object shown in the captured image isdetected, the image processing unit combines, with the captured image, avirtual object modified in accordance with the overlap.

(6) The image processing apparatus according to (5),

wherein the overlap of the object shown in the captured image and thepredetermined object is detected based on a position of a boundary linebetween the object shown in the captured image and the predeterminedobject.

(7) The image processing apparatus according to (1),

wherein, in a case where a type of the object shown in the capturedimage is detected, the image processing unit combines, with the capturedimage, a virtual object selected in accordance with the type of theobject shown in the captured image.

(8) The image processing apparatus according to (1),

wherein, in a case where a type of the object shown in the capturedimage is detected, the image processing unit combines a virtual objectwith the captured image, by modifying another object shown in thecaptured image into a shape corresponding to the type of the objectshown in the captured image.

(9) The image processing apparatus according to (1), further including

a detection unit which detects a state or a type of the object shown inthe captured image.

(10) The image processing apparatus according to (1), further including

a display control unit which controls a display unit in a manner that animage that has been processed by the image processing unit is displayedon the display unit.

(11) An image processing method including determining, based on a stateor a type of an object shown in a captured image, a virtual object to becombined with the captured image.

(12) A program for causing a computer to function as an image processingapparatus, the image processing apparatus including an image processingunit which combines a virtual object with a captured image,

wherein the image processing unit determines the virtual object based ona state or a type of an object shown in the captured image.

The present disclosure contains subject matter related to that disclosedin Japanese Priority Patent Application JP 2011-244377 filed in theJapan Patent Office on Nov. 8, 2011, the entire content of which ishereby incorporated by reference.

What is claimed is:
 1. An image processing method, comprising: detectinga first object from a captured image, wherein the first objectcorresponds to a first portion of a first living object; detecting asecond object from the captured image, wherein the second object isdifferent from the first object; detecting a position of a boundary linebetween the first object and the second object; detecting an overlapbetween the first object and the second object based on the detectedposition of the boundary line; generating a virtual object based on thedetected position of the boundary line; modifying the virtual objectbased on the detected overlap between the first object and the secondobject; and displaying a modified image that includes the modifiedvirtual object superimposed onto at least one portion of the secondobject.
 2. The image processing method according to claim 1, furthercomprising determining a shape of the virtual object based on thedetected position of the boundary line.
 3. The image processing methodaccording to claim 1, wherein the second object corresponds to a secondportion of the first living object, and the second portion is differentfrom the first portion of the first living object.
 4. The imageprocessing method according to claim 1, further comprising modifying asize of the virtual object based on the second object.
 5. The imageprocessing method according to claim 1, wherein the first object and thesecond object are wearable by the first living object.
 6. The imageprocessing method according to claim 1, wherein the second objectcorresponds to a portion of a second living object.
 7. An imageprocessing apparatus, comprising: a central processing unit (CPU)configured to: detect a first object from a captured image, wherein thefirst object corresponds to a first portion of a first living object;detect a second object from the captured image, wherein the secondobject is different from the first object; detect a position of aboundary line between the first object and the second object; detect anoverlap between the first object and the second object based on thedetected position of the boundary line; generate a virtual object basedon the detected position of the boundary line; modify the virtual objectbased on the detected overlap between the first object and the secondobject; and control a display screen to display a modified image thatincludes the modified virtual object superimposed onto at least oneportion of the second object.
 8. The image processing apparatusaccording to claim 7, wherein the CPU is further configured to determinea shape of the virtual object based on the detected position of theboundary line.
 9. The image processing apparatus according to claim 7,wherein the second object corresponds to a second portion of the firstliving object, and the second portion is different from the firstportion of the first living object.
 10. The image processing apparatusaccording to claim 7, wherein the CPU is further configured to modify asize of the virtual object based on the second object.
 11. The imageprocessing apparatus according to claim 7, wherein the first object andthe second object are wearable by the first living object.
 12. The imageprocessing apparatus according to claim 7, wherein the second objectcorresponds to a portion of a second living object.
 13. A non-transitorycomputer-readable medium having stored thereon computer-executableinstructions that, when executed by a processor, cause the processor toexecute operations, the operations comprising: detecting a first objectfrom a captured image, wherein the first object corresponds to a firstportion of a first living object; detecting a second object from thecaptured image, wherein the second object is different from the firstobject; detecting a position of a boundary line between the first objectand the second object; detecting an overlap between the first object andthe second object based on the detected position of the boundary line;generating a virtual object based on the detected position of theboundary line; modifying the virtual object based on the detectedoverlap between the first object and the second object; and controllinga display screen to display a modified image that includes the modifiedvirtual object superimposed onto at least one portion of the secondobject.
 14. The non-transitory computer-readable medium according toclaim 13, further comprising determining a shape of the virtual objectbased on the detected position of the boundary line.
 15. Thenon-transitory computer-readable medium according to claim 13, whereinthe second object corresponds to a second portion of the first livingobject, and the second portion is different from the first portion ofthe first living object.
 16. The non-transitory computer-readable mediumaccording to claim 13, further comprising modifying a size of thevirtual object based on the second object.
 17. The non-transitorycomputer-readable medium according to claim 13, wherein the first objectand the second object are wearable by the first living object.
 18. Thenon-transitory computer-readable medium according to claim 13, whereinthe second object corresponds to a portion of a second living object.